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Abstract-We have developed several tools to allow instructors 
analyzing students' interactions in a Course Management System. 
In this paper we describe a Systematic Usability Evaluation made 
to test the usability of one of these tools, based on cookies. The 
evaluations were based on task analysis and follow-up studies. In 
addition to regular observation and questionnaires, 3D video was 
used to provide the evaluator with more information about 
interactions. The use of this technology helped us in identifying 
functions and features that need to be improved. 
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I. INTRODUCTION 

Most Course Management Systems (CMS) lack versatile 
tracking functions, as shown in different studies [1, 2]. 
Sometimes, they provide information but it is not as accurate as 
teachers' demand, for instance, to use them to predict and 
prevent from possible students' dropping off or failings. 
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Fig. 1 Learning analytics systems to enhance CMS at the university 

We have been working on building better tools for 
analysing students' interactions with a CMS at our university 
[3]. They are based on four different approaches, namely: log 
analysis, cookies analysis, records of interactions of the 
identified students by login name and surveys. Fig. 1 shows the 
structure of the tracking system which is divided into four parts. 
Firstly, the web log analysis tool analyses log server files. It 
shows information to the user (i.e. the teacher), based on the 
assumption that accesses are impersonal, since the log files do 
not storage login data. Secondly, a Cookies Tracking Tool 
(CTT) allows us to storage the user accesses based on the use 
of cookies. Thirdly, a more subjective approach based on 
surveys has been developed. Finally, an ad-hoc Tracking Tool 
(ATT) that records all the user accesses identified by their 
login name is being developed. 



In this paper we focus on the CTT, a cookies-based analysis 
tool. The application collects data about users' interactions 
with the CMS and stores them in a data base. Afterwards, the 
tool allows teachers to display information about students' 
accesses in different graphical and tabular formats. 

The user interface of the CTT is shown in Figure 2. Notice 
that there are four different areas of the tool interface, 
depending on the data the user wants to obtain and are 
identified in the figure with a number. Number 1, "statistics by 
visits' number" area allows the teacher to obtain the number of 
accesses to a subject page or to a part of it grouped by different 
criteria and visualized in different chart or table formats. 
Number 2, the "most frequently followed paths" area offers a 
tree visualization of the sequence of clicks or accesses to pages 
that the user has followed most frequently. Number 3, the 
"number of visits per page" area displays all of the system 
pages ranked according to the number of accesses they had. 
Finally, number 4, the "user's statistics by" area offers several 
features of the computing and connection resources of students 
that the user/teacher can wish to consult. The information 
requested from the four areas can be constraint to ranks of data 
in the upper area of the tool, identified by number 5. Finally, 
complementary functionality regarding maintenance of users of 
the application, interactive help, summary of accesses to the 
CMS, disconnection, etc. can be selected in the left-hand side 
menu. 
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Fig. 2 Learning analytics tool main interface 

This application was plugged-in to our CMS and has been 
collecting data for five months about accesses of students to 
web sites of different courses. We offered the tool to some 
teachers, who reported us problems on understanding all of the 
features of the tool. Consequently, we developed a Systematic 



JITAE Vol. 1 No. 1 2012 PP. 35-39 www.jitae.org < 

-35- 



1 World Academic Publishing 



Journal of Information Technology and Application in Education 



(JITAE) 



Usability Evaluation (SUE) to enhance it. Our ultimate goal is 
to improve the tool interface before making it available to 
teachers for their dairy teaching activity. Of course, they had 
no previous experience in the use of the system. 

The paper describes the SUE Inspection we performed in 
our CTT. Section 2 describes the three parts of the SUE 
inspection. In the third and fourth sections, we describe the 
participants and the tasks they had to fulfil and an idea of the 
results they got. Section 5 contains results obtained from the 
three parts of the SUE. Finally, we summarize our conclusions. 

II. A SYSTEMATIC USABILITY EVALUATION 

There are many possible ways of combining the various 
usability methods [3] but it is highly recommended not to rely 
on a single one to the exclusion of others. 

Thus, we have carried out the SUE inspection by a 
combination of three usability methods, namely: observation, 
questionnaires, and interviews. 

The first method consists in the observation of the user as 
he/she works in the 10 different tasks the participants develop 
within the system. This method was improved by the use of 
video recording by using three cameras placed close to the user. 
The use of three video records allows the evaluator to obtain 
important usability information. The first one was focusing on 
his/her face, which gave the evaluator the impressions the 
different activities caused on his gestures or facial movements. 
The second camera was focusing on his/her computer screen, 
which allowed the evaluator to check what was exactly doing 
or trying to do a user in any moment of the SUE Inspection. 
This allowed observing particular interactions or substantial 
usability information. Finally, a third camera records the user 
environment, providing a view of the user interacting with the 
system, the mouse, visibility of the screen and general 
ergonomics the interface allows the user. All of this 
information provides important feedback to the evaluator about 
legibility, size, and user friendly characteristics of the 
application. Three snapshots of the three cameras are shown in 
Figure 3. 




Fig. 3 One moment of the evaluation with the three cameras: user, environment, 
and the screen 

The aim of the second method, fulfilling a usability 
questionnaire based on a set of heuristics, was to obtain more 
objective information about users' evaluations. 



Finally, the third method was an interview with users, 
where any lacks, dysfunctions, or extra functionality required 
was exposed as well as positive information about the usability 
he/she has felt. 

III. PARTICIPANTS 

Regarding the context of the evaluating tool, we found that 
users' experience in using this type of tools is fairly advanced 
and similar. 

We chose as evaluators teachers from different courses of 
Computer Science Engineering at the Universidad Rey Juan 
Carlos. They use a CMS as a platform to deliver courses 
material to their students. Thus, the information students 
retrieve is directly related to teaching pace in a given course. 
These evaluators have experience in using the CMS, but they 
have not used before the CTT whose usability we evaluated. 

As it is explained in [3], having a single evaluator enables 
us to find only 35% of the usability problems. However, since 
different evaluators tend to find different problems, it is 
possible to achieve substantially better performance by 
aggregating the results of several evaluators. The best rank is 
obtained by using five to tend evaluators. Thus, assuming 
Nielsen's 75% threshold in 5 evaluators and the same 
percentage for 7 evaluators using different techniques [5], we 
have used a total of 7 evaluators. 

IV. FULFILLING THE TASKS ON THE CTT 

In order to evaluate the cookies tool, we took into account 
the four dimensions that we found to be important as explained 
in [6], which are: 

• Presentation: all aspects bound to visualization of 
functions and elements of the cookies tool. 

• Hypermediality: aspects related to the way the user goes 
from one page to another, considering issues of quantity of 
memory required to fulfil the tasks and so on. Easiness of use. 

• Application proactivity: user's errors must be 
highlighted by the tool, allowing the user to "learn by doing" 
and by making errors. 

• User's activity: the user may need to perform unplanned 
activities. 

In our SUE Inspection we placed a set of ten tasks the user 
had to perform, as shown in Table 1. Reading left to right, the 
columns contain the task number, the task description, the way 
the query should be placed in the system by the user (i.e. its 
user interface), and finally, the query results. These tasks can 
be grouped in five groups. The first group includes tasks 1, 2, 
and 4, and deals with mundane tasks, such as creating and 
maintaining users into the system and consulting the online 
help. The second group is task 3 by itself, which offers a visual 
representation of the most commonly followed paths into the 
system composed of a given number of mouse clicks. The third 
group contains tasks 5 to 7, and permits the user to query about 
the number of visits to different subject pages in a selected 
time period. Information is grouped by different criteria and 
can be visualized in different graphical formats, including both 
chart and table representations. The fourth group consists in the 
task 8, which offers a visual representation of the number of 
visits per page. To conclude, the last group is formed by tasks 9 
and 10, which show users' statistics grouped by different 
selection criteria and for a certain period of time. 
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TABLE I 
ACTIVITIES CARRIED OUT BY THE PARTICIPANTS ON THE SUE INSPECTION 



10 



Task Description 



Access to the 
summary of visits 
the Tool provides 
as well as the 
online help 



Create a new user 
(teacher) in the 
system that would 
interact 



Consult the most 
common links 
followed by the 
user for 3 mouse 
clicks 



Change the 
password of a 
user 



Consult the 
statistics to a 
specific subject 
page in specific 
graphical and 
table format 
during November 
'06 



Consult the 
statistics in the 
day before to a 
subject page with 
another graphical 
representation 



Consults last 
week visits to a 
specific subject 
page with a pie 
chart and no table 



Consult number 
of visits per page 
during the month 
before 



Consult for a rank 
of data the origin 
of the visits 



Obtain for the 
current year the 
screen resolution 
used by the users 



Interface 
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All of the participants fulfilled satisfactorily all the tasks of 
the experiment, with little or no help at all, and in an average 
time of 36 minutes. Consequently, we think that the first 
usability inspection has resulted satisfactory; but let's check the 
rest of the results! 

V. RESULTS 

As some researches have shown [7], there are many task- 
specific and windows manipulation problems shown while 
observing the user or by interviewing him or her after 
performing tasks execution that would be missed by heuristics 
evaluation. This is the reason why the method of observation of 
the user as his/her work has resulted so effective. 

We explained here the results obtained by the different 
methods used. 

A. Observation 

As a result of observing users, as they worked with the 
system, we noticed some drawbacks that need to be improved, 
as well as some advantages that they enjoyed very much. Such 
features are summarized in Table 2. Although the precise 
symptom may not be illustrated, the reader can link the 
symptom to task number in Table 2. 

TABLE II USABILITY ERRORS IDENTIFIED BY THE OBSERVATION TESTING 



Symptom 


User Problem 


F 


HS 


Cause 


Re-entering of data 
when sending 
another query 

(task 5-10) 


Data reselected should 
stay when the user 
goes backwards 


6 


High 


OS functionality is 
not part of the user 
model 


Difficulty in finding 
the page that needs 
to be tracked 

(task 5-7) 


Site pages are not 
ordered alphabetically 


5 


Med 


Requirements 
specification did 
not include this 
feature 


Difficulty in finding 
the page that needs 
to be tracked 

(task 5-7) 


Necessity of creation 
of another search 
window for a specific 
page into the tool 


4 




The amount of 
web pages being 
tracked is very 
large 


User does not 
realize at first 
glance which part of 
the tool is using 

(task 5-10) 


The user forgets to 
mark the check box 
indicating in which 
part is working 


5 




The part of the 
tool that is being 
used should be 
highlighted 
automatically 


User does not see 
the global summary 
of access at first 
sight 

(task 1) 


User does not realize 
that the screen 
contains the summary 
of accesses 


3 




The summary of 
accesses title 
should be 
highlighted or 
colors changed 


User cannot see the 
name of the small 
pie pieces in the pie 
chart graph 

(task 9) 


The labels of the pie 
graph step one over 
the others when they 
are too little so the 
user cannot read them 


3 


Med 


Pie graphs must be 
rethought and 
rebuilt to avoid 
this malfunction 


There are parts of 
the tool that not 
permit to select 
another type of 


User wants to change 
the layout of the 
information using 
another type of graph 


2 




The best graph 
was selected at the 
requirements 
analysis step 
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graph 
(task 8-10) 


and can not do it. 








Queries of most 
common links 
followed by a 
number of clicks is 
too slow 

(task 3) 


User gets bored 
waiting for the query 
to show its results 


2 


Med 


The big amount of 
data the query 
needs must be 
calculated in batch 
process at night 


Query of most 
common links can 
not be selected by a 
range of data 

(task 3) 


User has to ask the 
query for the whole 
time period (this is 
why it is getting so 
slow) 


2 




Requirements did 
not include the 
selection of 
specific dates 


All the pages are 
shown in the query 
by number of users 

(task 9-10) 


There is no possibility 
to select a single page 
in the query, so it 
shows them all 


2 




Requirements did 
not include the 
selection of a 
single page 


There is too much 
information in the 
statistics interface 

(task 5-10) 


User get lost because 
the screen is full of 
information, he wants 
a 'selection statistics 
menu' 


2 


Med 


It was thought to 
be most useful this 
way 


Data selection is not 
filled by the user in 
a query 

(task 5-10) 


User forgets to enter a 
data or mark a period 


1 


Low 


Time period box is 
on the top. It can 
be changed 


Names in the page 
subject to be tracked 
are not understood 

(task 5-7) 


The names are 
shortened down 
because they are too 
long 


1 


Low 


Little labels 
should appear 
when positioning 
there the mouse 


The pie graph does 
not show the 'name' 
of the day 

(task 7) 


User has to guess with 
a calendar which week 
day it is 


1 


Low 


We did not 
thought it was 
important, but it is 


User can mark 
several parts of the 
tool, but only one 
result is displayed 
each time 

(task 5-10) 


The user marks 
different general parts 
of the tool (which are 
separated with lines, 
but does not realized 
of it) 


1 


Med 


The system should 
not allow the user 
to mark data in 
more than one part 


Can not view 
scrollable window 
contents 

(task 5-7) 


The scroll bar arrows 
move too fast 


1 


Med 


It is a standard 
feature that 
probably can be 
changed 


All the information 
can be seen by all 
authenticated users 

(task 5-10) 


Each user can see 
his/her subject 
tracking information 
as well as others' 


1 


Med 


It was thought 
more useful this 
way in the 
requirement phase 


The user does not 
see a message with 
his name once into 
the tool 

(task 1-10) 


User likes to know he 
is the one that is using 
the tool. The session 
must be personalized 
with a 'Welcome your 
name' message 


1 


Low 


It was not thought 
important at the 
beginning. Easy to 
implement in the 
next version 



B. Questionnaires 

Immediately after fulfilling all the tasks, the user had to 
answer a questionnaire. The questions and results are shown in 
Table 3. The analysis of these results gives us a general 
impression of their first feelings about the tool. The rest of the 
methods complement information gathered here. 

TABLE III Results of the Questionnaire 



Questions 


1 


2 


3 


4 


5 


no 




yes 


1, Does the system provide the precise 
information for you to do the tasks? 






2 


5 




2. Do you consider the system easy to use? 




1 


2 


4 




3. If you would have to analyse the CMS 
activity would you use this tool? 






1 


5 


1 


4. If you would have to analyse the activity 
of any other kind of web site, would you use 
this tool? 






1 


4 


2 


5. Would it be easy for you to get to be an 
expert in this tool? 








3 


4 


6. When you do a task, do you exactly know 
which the following task you should do is? 




1 


2 


4 




7. Is easy to change from one task to another 
without loosing yourself? 




1 


1 


2 


3 


8. Have you easily find the exact link to 
complete the required tasks? 




2 


1 


3 


1 


9. Have you felt comfortable using this tool? 


1 




4 




2 



TABLE IV RESULTS OF THE INTERVIEW 



Commentary 


Freq. 


Very good. I like it. Very intuitive. Well done. Very nice. 
Easy to use 


7 


Good statistics. Good Information. Good graphics 


3 


Very interesting most followed paths visualization 


1 


Very good time data selection interface 


1 


Information about country of origin, screen resolution and 
browser very useful 


1 


Very good it offers also table information easy to download 


1 


Very good the four type of graph 


1 



It is important to emphasize that it was the very first time 
the users used the cookies tool. On one hand, we wanted users 
who did not have experience about both the functionality it 
provides and how to use it. Thus, we could get the very first 
impression they had about the tool usability. On the other hand, 
having tested the tool with no previous experience from the 
users delays obtaining results about the tool aims. Although in 
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question number 5, the percentage 4 out of the 7 users have 
said that they find very easy to be an expertise in using this tool 
(marked 5 on the scale), which means that the usability test has 
been passed fairly well, even though there might be some 
commentaries being made during the interview afterwards to 
the user. 

C. Interviews 

In the interview with the user afterwards, commentaries 
were fairly positive, see Table 4. 

VI. CONCLUSIONS 

We are really satisfied with the SUE Inspection carried out. 
It provided us very useful information that it is not easy to 
gather for different reasons. Firstly, it is hard for the participant 
in an experiment to write things down (it is very time 
consuming). Secondly, the participant is not usually willing to 
offer all his/her impressions. Therefore, observation in the 
usability laboratory produced all of this information without 
much effort from the participant, simply by observing him/her 
by the evaluator as many times as needed. 

In addition, information gathered by questionnaires is good 
but not as detailed as that obtained from observation of the user. 
Finally, the interview reveals impressions about the tool as a 
whole that also are useful to complete the SUE Inspection. 
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